Skip to content

fix(client-runtime): returning to the app no longer restarts the reconnect backoff for an unreachable host - #24064

Open
Batuhan1999 wants to merge 1 commit into
pingdotgg:mainfrom
Batuhan1999:fix/ssh-wakeup-backoff
Open

Batuhan1999 wants to merge 1 commit into
pingdotgg:mainfrom
Batuhan1999:fix/ssh-wakeup-backoff

Conversation

@Batuhan1999

Copy link
Copy Markdown
Contributor

Problem

When a saved environment stays unreachable, the reconnect backoff from #14897 should climb to its 5-minute cap. In practice it rarely gets past a few seconds while the app is in use.

Real telemetry from one macOS desktop install (nightly 0.0.46-nightly.20261011). It had a saved SSH environment whose hostname no longer resolved. Between Oct 20 16:59 and Oct 23 11:33 local time:

  • desktop.ipc.sshEnvironment.ensureEnvironment failed 2,564 times (de-duplicated). All but one failure was ssh: Could not resolve hostname <host>.
  • The peak was 423 attempts in one hour.
  • The retry ladder reset about 370 times. A reset here means the gap between attempts fell back under 3 s after it had grown past 4 s, so the count is a heuristic.
  • The longest delay reached before a reset was 13 s at the median and 65 s at p90. The 300 s cap was almost never reached.

Mechanism on main:

  • apps/web/src/connection/platform.ts sends an application-active wakeup on every visibilitychange to visible.
  • resetsRetryBackoff in packages/client-runtime/src/connection/wakeups.ts returns true for every application-active variant and for network-changed.
  • In EnvironmentSupervisor.run, a wakeup during backoff ends the wait, and then resetRetryLadder() sets failureCount = 0. So each time the app comes back to the foreground, the next delays start again at ~1–2 s.

Change

This is a change to packages/client-runtime/src/connection/supervisor.ts only, so web, desktop, and mobile all pick it up.

  • A wakeup still ends the retry wait and starts an attempt immediately. Reconnecting as soon as you return to the app works as before.
  • An application-active wakeup now resets the retry ladder at most once per RETRY_MAX_DELAY_MS (5 minutes). The supervisor records when the last reset from a wakeup happened, using Clock. Within that window, a wakeup still causes an immediate attempt, but a failure afterwards continues up the ladder instead of starting again at the first rung.
  • network-changed still resets the ladder every time, because a network change can make the host reachable.
  • The same rule covers all three places where the run loop resets the ladder after a wakeup: backoff, offline, and blocked.
  • Mobile: application-active-probe and application-active-reconnect are handled the same way as desktop application-active. Each one means the user came back to the app, not that network conditions changed. A long mobile resume still replaces a live session, or interrupts an attempt in progress, and resets the ladder immediately (connectedLeaseEnd / waitForEstablishmentInterrupt). That path is not changed, because it handles sockets the OS killed in the background, not a host that stays down.
  • Error classification is not changed.

Trade-offs:

  • After 5 minutes or more without a reset, the next return to the app resets the ladder again. A host that stays down can still get a short burst of fast retries about once every 5 minutes.
  • Each return to the app still causes one immediate attempt. If the window becomes visible very often, those attempts happen just as often. The change removes the fast 2/4/8/16 s burst that followed each of them.

Scope and approval

Refs #15616. That issue is a triaged bug. The triage named "the reconnect backoff from #14897, especially how foreground and wakeup resets interact with it while a host stays unreachable" as the likely fix area. It also said that a maintainer will decide on the fix direction.

This PR proposes one direction: keep the immediate retry, and limit how often a wakeup can reset the ladder. Other options are to not reset on foreground at all, or to use a different window. Either would change only resetRetryLadderAfterWakeup and the test expectations.

Verification

New tests are in the existing packages/client-runtime/src/connection/supervisor.test.ts. They use it.effect with TestClock, and every attempt fails transiently ("Could not resolve hostname.").

  • retries at once on <reason> but keeps climbing the ladder, run once for each of application-active, application-active-probe, and application-active-reconnect. Four wakeups during backoff each start an attempt with no clock movement, and the backoff delays are [2000, 2000, 4000, 8000, 16000]. After TestClock.adjust("5 minutes"), the next wakeup resets the ladder again: attempt 1, 2000 ms.
  • restarts the retry ladder on every network change. After two application-active wakeups (attempt numbers 1, 1, 2), a network-changed wakeup brings the ladder back to attempt 1.
vp test run packages/client-runtime/src/connection/supervisor.test.ts -t "retries at once on|on every network change"
  • On main (test only, before the fix): 4 failed. The app-active tests got delays [2000, 2000, 2000, 2000, 2000] and the network-change test got [1, 1, 1, 1], because every wakeup reset the ladder. The app-active tests had a longer title during that run; the assertions were the same.
  • With the fix: 4 passed.

vp run check:changed upstream/main: format, lint, knip, typecheck (7 packages) and test (1 file, 34 tests) all pass.

Not checked:

  • No live run on desktop or mobile.
  • Wakeup reasons are not in telemetry. That the resets in the data came from application-active wakeups is an inference: they match the reset mechanism, and supervisor activity shows up between attempts.

Implemented with Claude Opus 5.5 via Claude Code (running in T3 Code).

🤖 Generated with Claude Code

…nnect backoff for an unreachable host

Every application-active wakeup ended the retry wait and reset the retry
ladder to its first rung. On desktop that fires on each visibilitychange to
visible, so a saved SSH environment whose host no longer resolved retried
every few seconds for as long as the app was in use and almost never
reached the 5 minute cap.

A wakeup still ends the wait and retries at once, but an application-active
wakeup (desktop or mobile) now resets the ladder at most once per
RETRY_MAX_DELAY_MS. A network change still resets it every time.

Refs pingdotgg#15616

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added vouch:unvouched Author is not yet trusted in the VOUCHED list. size:M 30-99 changed lines (additions + deletions). labels Oct 23, 2026
@macroscopeapp

macroscopeapp Bot commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Approved at b62097f

Macroscope's review found this PR approvable — This focused client-runtime fix prevents repeated foreground wakeups from restarting the reconnect backoff while preserving immediate retries and unconditional network-change resets. The added TestClock coverage verifies all wakeup variants and the five-minute reset behavior.

You can add or adjust custom eligibility rules. Learn more.

@coderabbitai

coderabbitai Bot commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
📥 Commits

Reviewing files that changed from the base of the PR and between d7a8d6f and b62097f.

📒 Files selected for processing: 2
  • packages/client-runtime/src/connection/supervisor.test.ts
  • packages/client-runtime/src/connection/supervisor.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 5 remain after this review.

📝 Walkthrough

The connection supervisor now uses wakeup reasons and elapsed time to determine when to reset its retry ladder. New tests cover retry timing and resets for application-active and network-changed wakeups.

Changes: Connection retry wakeup policy

Layer / File(s) Summary
Return wakeup reasons and apply reset rules
packages/client-runtime/src/connection/supervisor.ts
Retry and signal waits now return a wakeup reason or undefined. The reset helper always resets for network-changed and rate-limits other qualifying wakeups to once per maximum retry delay.
Integrate reset rules and test retry timing
packages/client-runtime/src/connection/supervisor.ts, packages/client-runtime/src/connection/supervisor.test.ts
Offline, blocked-state, and backoff waits pass wakeup reasons to the reset helper. Tests cover retry delays, reset timing, and attempt counts.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Suggested reviewers: t3dotgg

Merge Risk: ⚪ Minimal · up to b6209

The reconnect change appears ready to merge after normal checks; no actionable risk is established by the supplied evidence.

Pre-merge checks: Passed 4
Check name Status Explanation
Title check Passed The title is concise, specific, and accurately describes the primary reconnect backoff fix.
Description check Passed The description covers the problem, implementation, scope, approval reference, verification results, limitations, and agent details. It clearly explains the changed behavior and test evidence.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@lucastraba

Copy link
Copy Markdown

Another real-world case for this, on iOS, where the cost is higher than extra retries.

Setup: T3 Code iOS 2.0.0 (App Store), iOS 27.0, Tailscale iOS 1.104.4. Two environments: a Linux server that is always on, and a MacBook that is asleep most of the day. Both are reached over Tailscale.

What happened: while the Mac slept, the app kept retrying it (the UI kept flashing "reconnecting to "). On iOS, that traffic to an unreachable tailnet peer triggers tailscale/tailscale#21773: wireguard-go's 64-buffer pool fills up and the whole tunnel stalls for ~100 s. That includes the healthy Linux server, so the app lost its working environment as well, every 2–3 minutes.

Confirmed by the on/off switch from #11478: switching the Mac environment off stopped the stalls at once. There were none in the next 17 minutes of heavy use, against one every 2–3 minutes before. Over the following 9 days the Mac was offline in 165 hours, with the environment kept off, and the phone's tunnel restarted only twice.

The switch works, but only once you know the cause. A first-time user just sees T3 dropping its good environment. So I'd like to see this PR merged. A ladder that actually reaches its 5-minute cap on a host that stays down would keep the traffic below what tailscale/tailscale#21773 needs: per that issue, packets to the dead peer have to keep arriving more often than every ~90 s.

Related: #11236 (same report, closed after the on/off switch landed), tailscale/tailscale#21773.

@github-actions

Copy link
Copy Markdown
Contributor

This PR conflicts with main. Rebase it or merge main into it. If it still conflicts after 7 days with no activity, it will be closed automatically.

See the conflict window.

@mmmbuford

Copy link
Copy Markdown

The VPN toggle that recovers from tailscale/tailscale#21773 may also count as a network-changed, which this PR still resets on, so the offline host would go straight back to fast retries. Details in #37699 (comment).

Copy link
Copy Markdown

I’m seeing this repeatedly on T3 Code iOS 2.3.0 with two macOS hosts, both on 0.1.11 and paired via Tailscale HTTPS (Route: Direct). During the same incident, both hosts showed did not answer after two attempts. Both diagnostics showed This device: ok and /.well-known/t3/environment timed out after 10000ms.

This looks similar to #37699, but I haven’t yet checked whether Safari or another tailnet service also fails during an incident, or whether one host becoming unreachable triggers it. So I can’t tell whether this is the same tunnel-wide stall. Would an iOS Tailscale bug report ID and same-device Safari results captured during the next incident help distinguish that from an app-only reconnect issue?

@mmmbuford

Copy link
Copy Markdown

@mochafreddo Those symptoms match what I saw in #37699. The app said This device: ok, but a host that was up still got /.well-known/t3/environment timed out after 10000ms. These checks during the next incident would tell a tunnel-wide stall apart from an app-only reconnect problem:

  1. Safari on the same phone: open one of the hosts. If Safari hangs too, the whole Tailscale tunnel is stalled, not just T3's connections. That was my case.
  2. tailscale ping from one of the Macs to the phone every few seconds: in my case every ping failed until I toggled the VPN, even though the admin console still showed the phone as online.
  3. Any saved environment whose host was asleep or off just before it started: one unreachable host was enough to stall everything for me. Turning that environment off in the app stopped it right away.
  4. Tailscale iOS version: the stall is iOS: traffic to an unreachable peer exhausts the 64-buffer WireGuard pool and stalls all peers and 100.100.100.100 for ~100 s tailscale/tailscale#21773. Its fix was on TestFlight in 1.105.251, and the current stable release (1.104.7) doesn't have it yet.

A Tailscale bug report ID is most useful on tailscale/tailscale#21773, since only Tailscale can read those logs. The Safari and ping results are the useful part here, because they show whether keeping the backoff in this PR would be enough on its own.

Copy link
Copy Markdown

@mmmbuford Thanks for the detailed explanation and troubleshooting steps!

I'll check Safari connectivity and run tailscale ping the next time the issue occurs to determine whether it's a Tailscale tunnel issue or a T3 Code reconnect issue.

I'll also check my Tailscale iOS version and report back with the results.

@jieyuexing

Copy link
Copy Markdown
Contributor

[!NOTE] Drafted by Gemini 4 Neon in T3 Code; reviewed and posted by @jieyuexing

Adding another desktop data point for this fix.

Setup: macOS desktop 0.1.12-nightly.20261121.5064 (unmodified packages/client-runtime/src/connection/, up through #39431), with a saved remote environment reached through an on-demand HTTPS tunnel whose upstream VPN stays offline outside working hours.

Observed during a 3.7-hour offline window:

  • 80 connection attempts (~21.6/hour).
  • Across the 79 inter-attempt gaps: 55 landed in 150–310 s (reaching the 5-minute jittered cap), 7 in 60–150 s, and 17 were under 60 s right after returning to the desktop window (visibilitychange -> application-active resetting the retry ladder).

This also lines up with docs/internals/connection-runtime.md, which states that the jittered backoff "resets only after a connection stays up".

@Batuhan1999 Since this PR currently has needs-rebase (from the application-active-reconnect -> application-active-long rename in #25824 and the backoffFailures / approvalWaits split in #36419), do you plan to rebase it? If you're no longer maintaining it, I'm happy to open a clean replacement PR referencing #24064 so the fix doesn't get auto-closed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bugfix Fix PR opened by an issue lead; enters the PR loop needs-rebase Conflicts with its base branch. Closes after 7 days with no activity. pr:gate-failed A land gate failed size:M 30-99 changed lines (additions + deletions). vouch:unvouched Author is not yet trusted in the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants